1. Field of the Invention
The present invention relates to computer system security. More particularly, the present invention relates to a system and method of unpacking code that has been obfuscated with a packer on a computer system.
2. Description of the Related Art
Malicious code writers are increasingly using packers, such as run time packers, to obfuscate malicious code files and avoid detection of the malicious code files on computer systems. A packer is a tool that can be used to compress and/or obfuscate code, such as an executable malicious code file.
Typically a packer compresses, encrypts, or otherwise obfuscates, the code of an original executable file and appends on a small unpacking routine that is not packed. When the executable file starts up, the unpacking routine initially runs and decompresses the compressed original executable file in memory. When the unpacking routine is completed, the original executable file is in memory, e.g., as a memory image.
The process of packing a malicious code file changes the appearance of the malicious code file so that the malicious code file does not appear the same to an anti-virus (AV) engine as it did before. Thus, known malicious code signatures used by the AV engine will no longer match the packed malicious code file. Consequently, the packed malicious code file cannot be detected by the AV engine. When an AV engine cannot remove, or “unpack” the layers of obfuscation resulting from the packing process, a new malicious code signature needs to be created each time the malicious code file is re-packed.
Additionally, creating an unpacker, i.e., a program for unpacking the packed code, for use by an AV engine is a difficult task that currently requires an expert analyst to manually reverse-engineer the packing. Typically, each unpacker is unique and must be specifically created for each packer. This approach requires considerable time and effort, and can be very brittle. Malicious code writers can often avoid unpacking simply by changing a few bytes in the appended unpacking routine's start up code, so that the modified unpacking routine can no longer be identified as the original packer.
The rate at which malicious code writers can create packers exceeds the rate at which engineers can create unpackers, leading to a dangerous backlog of unsupported packer formats. Malicious code writers are aware of this difficulty and have begun employing an ever-growing number of packers to hide malicious code files and avoid detection by AV products.
Debugger-based unpacking is one example of a current unpacking technique that attempts to provide a more generic approach to unpacking packed files. In debugger-based unpacking, a breakpoint, or set of breakpoints, is placed on the expected entry point of an unpacked executable file. For executables running on WINDOWS systems, this is typically near 0x00400000. (WINDOWS is a registered trademark of Microsoft Corporation in the United States and other countries. A WINDOWS system is a computer system running a WINDOWS operating system.) Upon reaching the breakpoint, program execution is stopped and control of the program transferred to the debugger.
The program can be analyzed and/or scanned for malicious code. While debugger-based unpacking is typically suitable for certain older packer families, debugger-based unpacking has begun to lose its effectiveness. An increasing majority of newer packers employ anti-debug methods, such as debugger detection and exception handling, which cause debugger-based unpacking to fail. In such cases, the debugger may lose control of the program under analysis with unpredictable results.